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DETAILED ACTION 

Response to Amendment 
A request for continued examination under 37 CFR 1.114, including the fee set forth in 
37 CFR 1.17(e), was filed in this application after final rejection. Since this application is 
eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1 . 1 7(e) 
has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 
37 CFR 1.114. Applicant's submission filed 9/22/2006 has been entered. Claims 1, 10, 19, and 
31 have been amended. Claims 2-5, 1 1-14, 20-23, 30 and 32 have been canceled. Claims 1,6-10, 
15-19, 24-29 and 31 are pending in the application. 

Response to A rguments 

Applicant's arguments dated September 22, 2006, with respect to claims 1,6-10, 15-19, 
24-29 and 3 1 have been considered but are not found persuasive in view of the ground(s) of 
rejection set forth in the present Office Action based on Daily U.S. Patent No. 6,317,127 (Daily) 
in view of Ohshima et al. U.S. Patent No. 6,522,312 (Ohshima) and Zsolt Szalavari, Erik 
Eckstein, Michael Gervautz, "Collaborative Gaming in Augmented Reality", Proceedings of 
VRST*98, pp. 1 95-204, Taipei, Taiwan (hereinafter Szalavari). 

Although Daily does not expressly disclose to the claim limitation that "sensing a video 
of the real space including a player who is experiencing an augmented reality", Daily discloses 
displaying a sub-image of the wide FOV video signal to be distributed in real time over the 
network to a plurality of users 25 in response to the airplane's positioning information (column 
4, lines 15-20). Daily discloses that a user controllable viewpoint selector such as a head tracker 
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for tracking the user's head movement to select a user FOV from the wide FOV video signal 
(column 5, lines 1-20). In so doing, an image of the user is included in a video of the wide FOV 
captured by the fish eye lens 172 or geodesic array of sensors 170 (column 11-12). Therefore, 
Daily discloses a head tracker for recording the user's head movement, which is an image of the 
player/user that can be included in the user's FOV video for display on the user's HMD. 

Ohshima discloses a CCD camera 240 for capturing an image of the player into a pick-up 
image (e.g., column 18, lines 40-45), the so called marker image (column 13, lines 35-40) and 
the CCD camera 230 fixed above the center of the table can capture the entire surface of the table 
within its field of view in which multi-valued image data of the surface of the table sensed by 
the TV camera is acquired (wherein the TV camera captured images of the players as well 
as the virtual pluck and the virtual goal can be viewed by an observer on the TV camera 
230 and the camera's field of view selects which players and the virtual objects should be 
included within its field of view) and the CCD cameras 240 fixed to the heads of the players 
acquire marker images which are processed to detect positions of tracked markers falling within 
the respective fields of view of the individual cameras (column 15, lines 35-40 and column 15, 
lines 3-10). The two players can observe the surface of the table when they wear the HMDs. The 
players observe a three-dimensional image displayed on the display screen to be superposed on 
an image in the real space observed via optical system of the HMDs (column 12, lines 21-30). 
The changes in posture of the head of the player (an image of the player) are detected by the 
CCD camera 240 as changes in image sensed by the camera. In Fig. 7, the image as captured by 
the CCD cameras 240 can be included in a video sensed by the video camera 230 within the 
image generation units 5050L and 5050R. 
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Ohshima discloses seeing the game with the virtual pluck and the virtual goal along 
with the changes in image sensed by the CCD camera 240 in accordance with the changes 
in posture of the player's heads (See column 13, lines 15-30). 

Therefore, having the combined teaching of Daily and Ohshima, one of the ordinary skill 
in the art would be able to include an image of the player/user in a video of the table or the wide 
angle FOV. Doing so would allow the customized image generation for a particular player/user 
(Ohshima Fig. 7 and Daily column 5, lines 1-20). 

Daily and Ohshima do not expressly teach the claim limitation of "the observer" and the 
claim limitation of "first video composition means for composing an augmented reality video 
viewed from the objective viewpoint position on the basis of the sensed videos of the real space 
sensed by said first video sensing means and the generated video of the virtual object generated 
by said first video generation means." 

Ohshima implicitly discloses the claim limitation of the observer because Ohshima' s 
system of Figs. 7 and 23 includes the Mallet position measurement 5010, which comprises a 
computer system Onyx2 from Silicon Graphics, and Position/Posture Measurement 5000, which 
comprises a computer system Onyx2 from Silicon Graphics. The augmented reality video can be 
observed from the Onyx2 workstation by an observer. Therefore, Ohshima at least suggests the 
claim limitation of the observer. 

Szalavari expressly discloses the claim limitation of the observer in Fig.2 wherein the 
VR-hardware components seen from an observer is displayed (See Szalavari Section 3.2) 
including the superposition of the real object of the players and the virtual object of cards 
(Szalavari Section 4). Szalavari thus teaches the claim limitation of "first video composition 
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means for composing an augmented reality video viewed from the objective viewpoint position 
on the basis of the sensed videos of the real space sensed by said first video sensing means and 
the generated video of the virtual object generated by said first video generation means." 

Therefore, having the combined teaching of Daily, Ohshima and Szalavari, one of the 
ordinary skill in the art would be able to include an image of the player/user in a video of the 
table or the wide angle FOV. Doing so would allow the customized image generation for a 
particular player/user (Ohshima Fig. 7 and Daily column 5, lines 1-20 and Szalavari Fig. 2; 
Section 3.2 and 4). 

Applicants argue that "Applicant submits that an image obtained by composing an image 
like that of Fig. 2 and a CG, which appears a head in Szalavari with an indication 'view of a 
player', is not an image from an objective viewpoint" and "Applicants submit that such an image 
is merely an image from the viewpoint of another player". In response, Szalavari discloses that 
"Figure 2 shows our VR-hardware components as seen from an observer". It is clear from this 
teaching that the observer is not participated in the game and the viewed image in Fig. 2 is 
viewed by an observer as displayed in the tracker server or the game server for non-participants, 
as opposed to be displayed on the game players' HMD. Szalavari also discloses the claim 
limitation of the observer because Szalavari's system of Fig. 1 includes the game server and the 
tracker server which is a computer system having a display wherein the display displaying the 
game play is observed by the observer other than the players. The augmented reality video can 
be observed from the game server or the tracker server by an observer. Thus, applicant's 
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argument that such an image is merely an image from the viewpoint of another player is 
incorrect. 

Applicant further argues that Daily discloses that the camera is mounted on the underside 
or on top of an airplane, and therefore it captures an image of the external world of the airplane. 
In response, Daily discloses that a user controllable viewpoint selector such as a head tracker for 
tracking the user's head movement to select a user FOV from the wide FOV video signal 
(column 5, lines 1-20). In so doing, an image of the user is included in a video of the wide FOV 
captured by the fish eve lens 172 or geodesic array of sensors 170 (column 11-12). Therefore, 
Daily discloses a head tracker for recording the user's head movement, which is an image of the 
player/user that can be included in the user's FOV video for display on the user's HMD. 

Ohshima implicitly discloses the claim limitation of the observer because Ohshima' s 
system of Figs. 7 and 23 includes the Mallet position measurement 5010, which comprises a 
computer system Onyx2 from Silicon Graphics, and Position/Posture Measurement 5000, which 
comprises a computer system Onyx2 from Silicon Graphics. The augmented reality video can be 
observed from the Onyx2 workstation by an observer. Therefore, Ohshima at least suggests the 
claim limitation of the observer. 

Claim Rejections - 35 USC §101 
35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or 
any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and 
requirements of this title. 
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Claims 19 and 24-27 are rejected under 35 U.S.C. 101 because the claimed invention is 
directed to non-statutory subject matter. 

Claims 19 and 24-27: 

Claim 19 recites storage medium storing a program code. The claimed medium is not 
necessarily a computer readable medium. The claimed program code is not necessarily computer 
executable instructions. There is no structural and functional interrelationship between the 
software and the rest of the computer to permit the instructions' functionality to be realized. 
Claim 19 is, thus, non-statutory. 

Additionally, the claim 19 does not include a computer to execute the claimed program 
code, the claimed program code is merely a set of instructions capable of being executed by a 
computer, the claimed program code itself is not a process. Thus, claim 1 9 is also non-statutory 
for this reason. 

The claims 24-27 are subject to the same rationale of rejection set forth in the claim 19. 

Claim Rejections - 35 USC§103 
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 
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Claims 1, 6-10, 15-19, 24-29 and 31 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Daily U.S. Patent No. 6,317,127 (Daily) in view of Ohshima et al. U.S. Patent 
No. 6,522,312 (Ohshima) and Zsolt Szalavari, Erik Eckstein, Michael Gervautz, "Collaborative 
Gaming in Augmented Reality", Proceedings of VRST98, pp. 195-204, Taipei, Taiwan 
(hereinafter Szalavari). 

1. Claim 1: 

Daily teaches an augmented reality presentation apparatus for superimposing a virtual 
object in a real space, characterized by comprising: 

Objective viewpoint augmented reality presentation means for presenting an augmented 
reality view viewed from an objective viewpoint position, which differs from a viewpoint 
position of any player (e.g., figure 1, 9-10; column 3-5; column 10-12), 

wherein (Examiner notes: Language that suggests or makes optional but does not 
require steps to be performed or does not limit a claim to a particular structure does not limit the 
scope of a claim or claim limitation) said objective viewpoint augmented reality presentation 
means includes 

First video sensing means for sensing a video of the real space, including a player who is 
experiencing an augmented reality, viewed from the objective viewpoint position (e.g., CCD 
cameras or a multi-sensor system 152 attached to the underside of the airplane to image the 
hemisphere below the plane, the GPS 154 providing the plane 's current position in coordinates 
and a second sensor array placed on top of the airplane for viewing star constellations; figure 1, 
9-10; column 3-5; column 10-12. Daily discloses displaying a sub-image of the wide FOV video 
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signal to be distributed in real time over the network to a plurality of users 25 in response to the 
airplane 's positioning information (column 4, lines 15-20). Daily discloses that a user 
controllable viewpoint selector such as a head tracker for tracking the user 's head movement to 
select a user FOV from the wide FOV video signal (column 5, lines 1-20). In so doing, an image 
of the user is included in a video of the wide FOV captured by the fish eye lens 1 72 or geodesic 
array of sensors 1 70 (column 11-12). Therefore, Daily discloses a head tracker for recording 
the user 's head movement, which is an image of the player/user that can be included in the 
user 's FO V video for display on the user 's HMD)\ 

First video generation means for generating a video of the virtual object viewed from the 
objective viewpoint position {e.g., In column 3-4, the reference teaches that the augmented 
reality system includes a plurality of image sources generating respective video signals as 
sensed by the CCD cameras that sense real imagery and a wide FOV video signal is generated 
by the synthesizing the plurality of video signals and the positioning system provides the plane 's 
latitude, longitude and altitude and then the positioning information will be stored and 
registered to the signal. The generated video of the virtual sphere wherein the source processor 
16 includes individual channel processors 18 that group the pixels into packets and a multiplexer 
20 that multiplexes the packets into a wide FOV video signal 2, i.e.. video of real space, A source 
processor 16 includes individual channel processors 18 that 2roup the pixels into packets and a 
multiplexer 20 that multiplexes the packets into a wide FOV video signal 22 and the overlay 
distribution processor 24 interacts with the positioning system 32, the database 30 to determine 
which overlay symbols and audio tracks lie within the current wide FOV and the subimage 
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capture unit 50 downloads the data from the view and overlay distribution buses 24 and 26 and 
stores it in a buffer; figure 1, 9-10; column 3-5; column 10-12); 

First video composition means for composing an augmented reality video viewed from 
the objective viewpoint position on the basis of the sensed videos of the real space sensed by said 
first video sensing means and the generated video of the virtual object generated by said first 
video generation means (e.g., In column 3-4, the reference teaches that the augmented reality 
system includes a plurality of image sources generating respective video signals as sensed by the 
CCD cameras that sense real imagery and a wide FOV video signal is generated by the 
synthesizing the plurality of video signals and the positioning system provides the plane 's 
latitude, longitude and altitude and then the positioning information will be stored and 
registered to the signal to produce the virtual sphere . The generated video of the virtual sphere 
wherein the source processor 16 includes individual channel processors 18 that group the pixels 
into packets and a multiplexer 20 that multiplexes the packets into a wide FOV video signal 2, 
i.e., video of real space, and the overlay distribution processor 24 interacts with the positioning 
system 32, the database 30 to determine which overlay symbols and audio tracks, i.e., video of 
virtual space, lie within the current wide FOV and the subimage capture unit 50 downloads the 
data from the view and overlay distribution buses 24 and 26 and stores it in a buffer; figure 1, 9- 
10; column 3-5; column 10-12), and 

Objective viewpoint video display means for displaying the composed augmented reality 
video composed by said first video composition means on a screen of an observer's display 
apparatus, the observer's display apparatus being separate from a head-mounted display worn by 
any player (e.g., the multiplexer 20 that multiplexes the packets into a wide FOV video signal 22 
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and the overlay distribution processor 24 interacts with the positioning system 32, the database 
30 to determine which overlay symbols and audio tracks lie within the current wide FOVand the 
subimage capture unit 50 downloads the data from the view and overlay distribution buses 24 
and 26 and stores it in a buffer; A viewpoint display unit 52 reads out the video and overlay 
data from the buffer for the current user FOVand transmits the subimage to the HMD wherein 
the display 52 t which meets the claim limitation of "an observer 's display apparatus " is 
separate and different from the display within HMD; see figure 1, 9-10; column 3-5; column 10- 
12; The video signal 22, including the video of real space and the video of virtual space, is 
broadcast over a video distribution channel 24 such as a network to a plurality of users 25); 

wherein (Examiner notes: Language that suggests or makes optional but does not 
require steps to be performed or does not limit a claim to a particular structure does not limit the 
scope of a claim or claim limitation) said apparatus further comprises: 

Player's viewpoint augmented reality presentation means for presenting an augmented 
reality view viewed from a player's viewpoint (e.g., figure 1, 9-10; column 3-5; column 10-12); 

wherein (Examiner notes: Language that suggests or makes optional but does not 
require steps to be performed or does not limit a claim to a particular structure does not limit the 
scope of a claim or claim limitation) said player's viewpoint augmented reality presentation 
means includes 

A head-mounted display having a screen (e.g., HMD for displaying the user 's local FOV; 
figure 1, 9-10; column 3-5; column 10-12); 

Second video sensing means for sensing a video of the real space viewed from the 
player's viewpoint position (e.g., an external stereo camera setup slaved to the user 's head 
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movement so that the cameras track his head movement and provide the desired view of the 
surroundings; column 1; and the tracking device 164 such as an RF beacon or a gyro compass 
to follow the user 's head movement and output coordinates that define the users FOV, or the 
viewpoint position; figure 1, 9-10; column 3-5; column 10-12); 

Player's viewpoint position acquiring means for acquiring information indicating the 
player's viewpoint position (e.g., an external stereo camera setup slaved to the user's head 
movement so that the cameras track his head movement and provide the desired view of the 
surroundings; column 1; and the tracking device 164 such as an RF beacon or a gyro compass 
to follow the user's head movement and output coordinates that define the users FOV, or the 
viewpoint position; figure 1, 9-10; column 3-5; column 10-1 2); 

Second video generation means for generating a video of the virtual object viewed from 
the player's viewpoint position using the information indicating the player's view position (e.g., 
the distribution processor 24 uses the position and heading information including the 
appropriate geographical range, class and hierarchical levels of symbols and audio tracks for 
each user to obtain a subimage 42 of the wide FOV video signal 22; figure 1, 9-10; column 3-5; 
column 10-12); 

Second video composition means for composing an augmented reality video viewed from 
the player's viewpoint position on the basis of the sensed videos of the real space sensed by said 
second video sensing means and the generated video of the virtual object generated by said 
second video generation means (e.g., in response to user 's FOV user 's focal length, user 's GPS 
coordinates, etc., column 11. wherein the player's viewpoint position is determined, each 
passenger wears a HMD to view a subimage 161 of the wide FOV video signal and the 
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passengers look around inside the virtual sphere to select and view a certain portion of the wide 
FOV. A zoom controller 168 allows the passenger to zoom in and out on the video signal 
Moreover, the distribution processor 24 uses the position and heading information including the 
appropriate geographical range, class and hierarchical levels of symbols and audio tracks for 
each user to obtain a subimage 42 of the wide FOV video signal 22; figure 1, 9-10; column 3-5; 
column 10-12)\ and 

Player's viewpoint video display means (e.g., HMD for displaying the user 's local FOV; 
figure 1, 9-10; column 3-5; column 10-12) for displaying the composed augmented reality video 
composed by said second video composition means on the screen of said head-mounted display 
(e.g., in response to user's^ FOV user's focal lemth user's GPS coordinates, etc., column 77. 
wherein the player } s viewpoint position is determined, each passenger wears a HMD to view a 
subimage 161 of the wide FOV video signal and the passengers look around inside the virtual 
sphere to select and view a certain portion of the wide FOV. A zoom controller 168 allows the 
passenger to zoom in and out on the video signal Moreover, the distribution processor 24 uses 
the position and heading information including the appropriate geographical range, class and 
hierarchical levels of symbols and audio tracks for each user to obtain a subimage 42 of the 
wide FOV video signal 22; figure 1, 9-10; column 3-5; column 10-12), wherein said head- 
mounted display includes said second video sensing means (e.g., an external stereo camera setup 
slaved to the user 's head movement so that the cameras track his head movement and provide 
the desired view of the surroundings; column 1; and the tracking device 164 such as an RF 
beacon or a gyro compass to follow the user 's head movement and output coordinates that 
define the users FOV, or the viewpoint position; figure 1, 9-10; column 3-5; column 10-12) and 
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said player's viewpoint video display means, and is worn by a player (e.g., HMD for displaying 
the user's hcalFOVj figure 1, 9-10; column 3-5; column 10-12). 

Although Daily does not expressly disclose to the claim limitation that "sensing a video 
of the real space including a player who is experiencing an augmented reality", Daily discloses 
displaying a sub-image of the wide FOV video signal to be distributed in real time over the 
network to a plurality of users 25 in response to the airplane's positioning information (column 
4, lines 15-20). Daily discloses that a user controllable viewpoint selector such as a head tracker 
for tracking the user's head movement to select a user FOV from the wide FOV video signal 
(column 5, lines 1-20). In so doing, an image of the user is included in a video of the wide FOV 
captured by the fish eye lens 172 or geodesic array of sensors 170 (column 11-12). Therefore, 
Daily discloses a head tracker for recording the user's head movement, which is an image of the 
player/user that can be included in the user's FOV video for display on the user's HMD. 

Ohshima discloses a CCD camera 240 for capturing an image of the player into a pick-up 
image (e.g., column 18, lines 40-45), the so called marker image (column 13, lines 35-40) and 
the CCD camera 230 fixed above the center of the table can capture the entire surface of the table 
within its field of view in which multi-valued image data of the surface of the table sensed by 
the TV camera is acquired (wherein the TV camera captured images of the players as well 
as the virtual pluck and the virtual goal can be viewed by an observer on the TV camera 
230 and the camera's field of view selects which players and the virtual objects should be 
included within its field of view) and the CCD cameras 240 fixed to the heads of the players 
acquire marker images which are processed to detect positions of tracked markers falling within 
the respective fields of view of the individual cameras (column 15, lines 35-40 and column 15, 
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lines 3-10). The two players can observe the surface of the table when they wear the HMDs. The 
players observe a three-dimensional image displayed on the display screen to be superposed on 
an image in the real space observed via optical system of the HMDs (column 12, lines 21-30). 
The changes in posture of the head of the player (an image of the player) are detected by the 
CCD camera 240 as changes in image sensed by the camera. In Fig. 7, the image as captured by 
the CCD cameras 240 can be included in a video sensed by the video camera 230 within the 
image generation units 5050L and 5050R. 

Ohshima discloses seeing the game with the virtual pluck and the virtual goal along 
with the changes in image sensed by the CCD camera 240 in accordance with the changes 
in posture of the player's heads (See column 13, lines 15-30). 

Therefore, having the combined teaching of Daily and Ohshima, one of the ordinary skill 
in the art would be able to include an image of the player/user in a video of the table or the wide 
angle FOV. Doing so would allow the customized image generation for a particular player/user 
(Ohshima Fig. 7 and Daily column 5, lines 1-20). 

Daily and Ohshima do not expressly teach the claim limitation of "the observer" and the 
claim limitation of "first video composition means for composing an augmented reality video 
viewed from the objective viewpoint position on the basis of the sensed videos of the real space 
sensed by said first video sensing means and the generated video of the virtual object generated 
by said first video generation means." 

Ohshima implicitly discloses the claim limitation of the observer because Ohshima' s 
system of Figs. 7 and 23 includes the Mallet position measurement 5010, which comprises a 
computer system Onyx2 from Silicon Graphics, and Position/Posture Measurement 5000, which 
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comprises a computer system Onyx2 from Silicon Graphics. The augmented reality video can be 
observed from the Onyx2 workstation by an observer. Therefore, Ohshima at least suggests the 
claim limitation of the observer. 

Szalavari expressly discloses the claim limitation of the observer in Fig.2 wherein the 
VR-hardware components seen from an observer is displayed (See Szalavari Section 3.2) 
including the superposition of the real object of the players and the virtual object of cards 
(Szalavari Section 4). Szalavari thus teaches the claim limitation of "first video composition 
means for composing an augmented reality video viewed from the objective viewpoint position 
on the basis of the sensed videos of the real space sensed by said first video sensing means and 
the generated video of the virtual object generated by said first video generation means." 

Therefore, having the combined teaching of Daily, Ohshima and Szalavari, one of the 
ordinary skill in the art would be able to include an image of the player/user in a video of the 
table or the wide angle FOV. Doing so would allow the customized image generation for a 
particular player/user (Ohshima Fig. 7 and Daily column 5, lines 1-20 and Szalavari Fig. 2; 
Section 3.2 and 4). 

Claim 6: 

The apparatus according to claim 7, characterized in that parameters of said first video 
sensing means are known, and said first video generation means generates the video of the 
virtual object viewed from said first viewpoint position in accordance with the known 
parameters. 
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Claim 6 recites all the limitations of claim 1 and adds the limitation of "the known 
parameters." Daily teaches the known parameters such as the focal length of the CCD video 
camera and measured plane's coordinates by GPS sensors, and input parameters such as the 
desired resolution, the object's focal length, orientation and Cartesian coordinates on the object's 
image plane where the object is either the image source or the user display {figure 1, 9-10; 
column 3-5; column 7-12), 

Claim 7: 

The apparatus according to claim 1, characterized in that some of parameters of said 
first video sensing means are variable, said apparatus further comprises measurement means for 
measuring changes of the parameters, and said objective video generation means generates the 
video of the virtual object viewed from said first viewpoint position in accordance with the 
parameters measured by said measurement means. 

Claim 7 recites all the limitations of claim 1 and adds the limitation of "variable 
parameters" and "measurement means." Daily teaches measuring the plane's positions are 
variables of the plane's coordinates in which the coordinate values are measured by the GPS 
sensors {figure 1, 9-10; column 3-5; column 7-12), 

Claim 8: 

The apparatus according to claim 7, characterized in that the parameters of said first 
video sensing means measured by said measurement means include at least one of a viewpoint 
position/posture, and zoom ratio. 
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Claim 8 recites all the limitations of claim 7 and adds the limitation of "position/posture 
information." Daily teaches the plane's positions are variables of the plane's coordinates which 
determine the position/posture of the CCD video cameras and the zoom ratios are determined by 
the zooming in and out parameters inputted by the user. Daily also teaches generating and 
correcting view position and posture of the passengers {figure I, 9-10; column 3-5; column 7- 
72). 

Claim 9: 

The apparatus according to claim 1, characterized in that when a plurality of first video 
sensing means equivalent to said first video sensing means are present, said apparatus further 
comprises selection means for receiving a plurality of videos of the real space from said 
objective viewpoint position from the plurality of first video sensing means, and outputting a 
video of the real space viewed from said objective viewpoint position from one selected first 
video sensing means to said first video composition means, and said first video composition 
means generates a video of the virtual object viewed from said objective viewpoint position using 
parameters of the first video sensing means selected by said selection means. 

Claim 9 recites all the limitations of claim 1 and adds the limitation of "selection means 
for receiving a plurality of first video sensing means." 

Daily teaches selecting of the image regions as captured by a plurality of video cameras 
by zooming in and out the video scene in the real space. Daily also teaches a plurality of video 
cameras for sensing a scene below or around the plane where image signals that represent an 
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environment scene of a real space and the user's zooming in and out video scene {figure 1, 9-10; 
column 3-5; column 7-12). 



2. Claim 10: 

The claim 10 is a rephrasing of the claim 1 in a method form. The claim is rejected for 
the same reason as set forth in claim 1 . 

Claim 15: 

The method according to claim 10, characterized in that parameters of said first video 
sensing means are known, and said first video generation step includes the step of generating the 
video of the virtual object viewed from said objective viewpoint position in accordance with the 
known parameters. 

Claim 15 recites all the limitations of claim 10 and adds the limitation of "the known 
parameters." Daily teaches the known parameters such as the focal length of the CCD video 
camera and measured plane's coordinates by GPS sensors, and the video scenes can be adjusted 
in accordance to the desired resolution, the object's focal length, orientation and Cartesian 
coordinates on the object's image plane where the object is either the image source or the user 
display (figure 1, 9-10; column 3-5; column 7-12). 



Claim 16: 
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The method according to claim 10, characterized in that some of parameters of means for 
sensing a video viewed from said first viewpoint position are variable, said method further 
comprises the measurement step of measuring changes of the parameters, and said first video 
generation step includes the step of generating the video of the virtual object viewed from said 
objective viewpoint position in accordance with the parameters measured in the measurement 
step. 

Claim 16 recites all the limitations of claim 10 and adds the limitation of "variable 
parameters" and "measurement step." Daily teaches measuring the plane's coordinates which are 
variables as plane moves. Daily further teaches correcting view position and posture of the 
passengers and zooming in and out the video scene (figure 1, 9-10; column 3-5; column 7-12). 

Claim 17: 

The method according to claim 16, characterized in that the parameters of the means for 
sensing a video viewed from said objective viewpoint position measured in the measurement step 
include at least one of a viewpoint position/posture, and zoon ratio. 

Claim 17 recites all the limitations of claim 16 and adds the limitation of 
"position/posture information." Daily teaches the plane's positions are variables of the plane's 
coordinates which determine the position/posture of the CCD video cameras and the zoom ratios 
are determined by the zooming in and out parameters inputted by the user. Daily also teaches 
generating and correcting view position and posture of the passengers {figure 1, 9-10; column 3- 
5; column 7-12). 
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Claim 18: 

The method according to claim 10, characterized in that when a plurality of means for 
sensing a video viewed from said objective viewpoint position are present, said method further 
comprises the selection step of receiving a plurality of videos of the real space viewed from an 
objective viewpoint position from the plurality of means for sensing a video viewed from said 
objective viewpoint position, and outputting the video of the real space viewed from an objective 
viewpoint position input from one selected means for sensing a video of said objective viewpoint 
position to means for compositing an objective viewpoint video, and said first video composition 
step includes the step of generating a video of the virtual object viewed from said objective 
viewpoint position using parameters of the means for sensing a video viewed from an objective 
viewpoint position selected in the selection step. 

Claim 18 recites all the limitations of claim 10 and adds the limitation of "selection step 
for receiving a plurality of videos." 

Daily teaches selecting of the image regions as captured by a plurality of video cameras 
by zooming in and out the video scene in the real space. Daily also teaches a plurality of video 
cameras for sensing a scene below or around the plane where image signals that represent an 
environment scene of a real space and the user's zooming in and out video scene {figure 1, 9-10; 
column 3-5; column 7-12). 



3. Claims 28-29: 
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Referring to claims 28-29, Daily discloses a mixed reality presentation system that 
generates and presents a virtual image in a real space {figure 1, 9-10; column 3-5; column 10- 
12). However, the reference is silent on a printing means in connection to the mixed reality 
presentation apparatus. It is common that a computer system has a printing means attached to 
them. Therefore, it would have been obvious to one having ordinary skill in the art to have 
incorporated a printing means in the mixed reality presentation apparatus of Daily because such 
construction is rather conventional. A person of ordinary skill in the art would be motivated to 
have incorporated a printing means to the Daily's mixed reality presentation apparatus to further 
provide a paper copy of still images of the real images that have been displayed on the display 
devices of Daily. 

4. Claim 31: x 

The claim 3 1 encompasses the same scope of invention as that of claim 1 . The claim is 
subject to the same reasons set forth in claim 1 . 

Claims 19 and 24-27 rejected under 35 U.S.C. 103(a) as being unpatentable over Daily 
U.S. Patent No. 6,317,127 (Daily) in view of Ohshima et al. U.S. Patent No. 6,522,312 
(Ohshima) and Zsolt Szalavari, Erik Eckstein, Michael Gervautz, "Collaborative Gaming in 
Augmented Reality", Proceedings of VRST'98, pp. 195-204, Taipei, Taiwan (hereinafter 
Szalavari), Latypov U.S. Patent No. 6,624,853 (Latypov) and Sato U.S. Patent No. 6,445,815 
(hereinafter Sato). 
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5. Claim 19: 

(1) The claim 19 encompasses the same scope of invention as that of claim 1 except 
additional claimed limitation of "a storage medium storing a program code. 

(2) It is unclear whether Daily, Ohshima and Szalavari have implicitly taught a storage 
medium storing a program code. 

(3) However, Sato discloses the claimed limitation of a storage medium that stores an 
image processing program, which is implemented on a computer and continuously presents 
three-dimensional images to an observer/player, storing a program code of an augmented reality, 
a depth estimation program code, a depth image generation program code, a position/posture 
information estimation program code, a warping program code and a program code of presenting 
to the observer three-dimensional images. The Sato reference implicitly teaches a program code 
including the augmented reality presentation step of generating and presenting a virtual image in 
a real space in which an image of merged real objects and virtual objects are displayed in LCDs 
103 in an optical see-through head mounted device (column 1, lines 13-23). The Sato reference 
also implicitly teaches a position/posture estimation module 201 that outputs three-dimensional 
motions from a viewpoint position of the camera to right and left viewpoint positions of the 
player (column 12, lines 54-65). The Sato reference further teaches a depth image generation 
module 300 that uses position/posture information input from the position/posture estimation 
module 201 as that for CG rendering and which generates an augmented reality image using the 
three-dimensional CG database in accordance with the distance to an object in the real world 
expressed by the depth image and presents it on the LCDs 103 (column 12, lines 44-50) and a 
depth warping module 203 to inversely project a depth image ID acquired at a viewpoint having 
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position/posture information into a space, and to re-project it onto the imaging plane of the 
virtual camera with the focal length of the virtual camera assumed at the viewpoint having an 
estimated position/posture value by the viewpoint position/posture module 201 (column 13, lines 
32-48). 

(4) Daily at least suggests program codes illustrated in a variety of flow charts (Figs. 5- 

9). 

(5) One having the ordinary skill in the art would have been motivated to do this because 
it would have provided the installation and implementation of the image processing on other 
systems. 

Claim 24: 

The medium according to claim 19, characterized in that parameters of means for 
sensing said objective viewpoint video are known, and the program code of said first video 
generation step includes the step of generating the video of the virtual object viewed from said 
objective viewpoint position in accordance with the known parameters. 

Claim 24 recites all the limitations of claim 19 and adds the limitation of "the known 
parameters." Daily teaches the known parameters such as the focal length of the CCD video 
camera and measured plane's coordinates by GPS sensors, and the video scenes can be adjusted 
in accordance to the desired resolution, the object's focal length, orientation and Cartesian 
coordinates on the object's image plane where the object is either the image source or the user 
display {figure 1, 9-10; column 3-5; column 7-12). 
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Claim 25: 

The medium according to claim 19, characterized in that some of parameters of means 
for sensing a video viewed from said objective viewpoint position are variable, the program code 
of said medium further comprises the measurement step of measuring changes of the parameters, 
and the program code of said first video generation step includes the step of generating the video 
of the virtual object viewed from said objective viewpoint position in accordance with the 
parameters measured in the measurement step. 

Claim 25 recites all the limitations of claim 19 and adds the limitation of "variable 
parameters" and "measurement step." Daily teaches measuring the plane's coordinates which are 
variables as plane moves. Daily further teaches correcting view position and posture of the 
passengers and zooming in and out the video scene {figure 1, 9-10; column 3-5; column 7-12). 

Claim 26: 

The medium according to claim 25, characterized in that the parameters of the means of 
sensing a video viewed from said objective viewpoint position measured in the measurement step 
include at least one of a viewpoint position/posture, and zoon ratio. 

Claim 26 recites all the limitations of claim 25 and adds the limitation of 
"position/posture information." Daily teaches the plane's positions are variables of the plane's 
coordinates which determine the position/posture of the CCD video cameras and the zoom ratios 
are determined by the zooming in and out parameters inputted by the user. Daily also teaches 
generating and correcting view position and posture of the passengers {figure 1, 9-10; column 3- 
5; column 7-12). 
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Claim 27: 

The medium according to claim 19, characterized in that when a plurality of means for 
sensing a video viewed from said objectivet viewpoint position are present, said medium further 
comprises a program code of the selection step of receiving a plurality of videos of the real 
space viewed from an objective viewpoint position from the plurality of means for sensing a 
video viewed from said objective viewpoint position, and outputting the video of the real space 
viewed from an objective viewpoint position input from one selected means for sensing a video of 
said objective viewpoint position to means for compositing an objective viewpoint video, and the 
program code of said first video composition step includes the step of generating a video of the 
virtual object viewed from said objective viewpoint position using parameters of the means for 
sensing a video viewed from an objective viewpoint position selected in the selection step. 

Claim 27 recites all the limitations of claim 19 and adds the limitation of "selection step 
for receiving a plurality of videos." 

Daily teaches selecting of the image regions as captured by a plurality of video cameras 
by zooming in and out the video scene in the real space. Daily also teaches a plurality of video 
cameras for sensing a scene below or around the plane where image signals that represent an 
environment scene of a real space and the user's zooming in and out video scene {figure 1, 9-10; 
column 3-5; column 7-12). 



Conclusion 
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Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jin-Cheng Wang whose telephone number is (571) 272-7665. 
The examiner can normally be reached on 8:00 - 6:30 (Mon-Thu). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Kee Tung can be reached on (571) 272-7794. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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